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ABSTRACT 

Technology, which forms part of all educational settings and curricula, is becoming an integral part 
of higher education too. Therefore, observations and interpretations from inside such settings are 
needed for a more realistic understanding. This study evaluates the information packages of the 
undergraduate programs of a university within the framework of a digital university, using program 
evaluation and document review’ of written curricula. The “Educational Connoisseurship and 
Criticism Model” was used for curriculum evaluation. The information packages of the 
undergraduate programs were generally found to reflect specific intentions concerning technology 
integration. However, these intentions have not yet become manifest in the learning outcomes of the 
programs. Thus, this study shows that, although technology has entered into all fields and 
programs, at least regarding learning outcomes, it has not found inadequate reciprocity within the 
processes of teaching and learning. 
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INTRODUCTION 



esearch indicates that today’s technologies offer powerful capabilities for creating high-quality 
learning resources, such as visualization, simulation, games, interactivity, intelligent tutoring, 
collaboration, assessment and feedback (Expanding Evidence Approaches for Learning in a Digital 
World U.S. Department of Education Office of Educational Technology Report, 2013, p.7) Within the last decade, 
universities have experienced significant phases changes in making technology a transformative part of education, 
including efforts organize distance and online education (Parker, Lenhart, Moore, 2011). The effectiveness of a 
university can be evaluated from several program (for instance, its contribution to industry, developing creative 
individuals, contributing to political and social life). Major universities and research institutes in many countries are 
currently evaluating the effectiveness of their investments in technology. However, Keller argues that ‘the larger 
question is whether the impressive investments in people will be sufficient to propel these countries to a global 
leadership position in innovations and economic development based on information technology’ (2008 p. 101), 
suggesting that expectations concerning higher education have been rapidly changing in the last two decades. 


For many higher education programmes, extending learning opportunities so that they better meet learners’ 
and employers’ needs remains a major challenge. Redesigning learning to address the needs of lifelong learners has 
implications for pedagogy, business processes and information technology systems (JISC, Institutional Innovation 
Programme, cited Drysdale, 2012) Higher education has already experienced significant change driven by digital 
technology, resulting in heavy dependence on this technology. Research and scholarship are also highly dependent 
on information technology: for example, the use of computers to simulate physical phenomena, networks to link 
investigators in virtual laboratories or ‘collaboratories’, and digital libraries to provide scholars with access to 
knowledge resources (Duderstadt, 2003, p.39). All these efforts may be considered as a prerequisite for developing 
students as individuals with digital skills, and the capability to use digital skills flawlessly has recently become the 
primary quality sought in students graduating from universities. Possession of digital skills is considered as a 
prerequisite for innovation, creativity and efficiency in many industries. Universities are therefore trying to prepare 
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their curricula on the basis of these skills, with technology and consequently digital skills no longer being 
considered as mere tools of education but important outcomes. 

By ‘digital literacy’ we include skills for reading and interpreting the media in order to reproduce verbal 
and visual information, and the implementation and evaluation of new information received from the digital 
environment (Jones-Kavalier and Flannigan, 2006). Thus, the Educational Testing Service of America (2002) 
proposes the concept of ‘digital competence’. Here, digital competence is defined as a combination of information, 
skills and attitudes for the safe and critical use of information society technologies for work, entertainment and 
communication purposes (Jun and Fun, 2011 p. 59). As part of the curriculum, developing digital literacy is not 
about being fashionable or merely trying to engage students in learning; it is about addressing the changing nature of 
subject knowledge and acknowledging that young people are going to need different kinds of skills, knowledge and 
understanding in order to develop their expertise. That is, developing digital literacy in subject teaching enables 
young people to be effective, competent, critical students of that subject in the digital age (Hauge and Payton, 2010, 
p. 12). “When technology is integrated throughout the curriculum, students will not only learn technology skills but 
also content knowledge” (Frei, Gammill and Irons, 2009). 

If provision of support for digital literacy has now become a requirement for university curricula then it is 
necessary to ensure that the standards that constitute the concept of digital literacy are discussed within a clear 
framework so that curricula can be configured according to this framework. The literature on 21st century skills 
includes various frameworks, such as the Partnership for the 21st Century Skills Framework (2006), EnGauge 
Framework from Metiri/NCREL and the framework of American Association of College and Universities (Dede, 
2010). In general terms, they share the concept of digital literacy. According to Kuo and others (2012), 
“Technological literacy involves a vision where each citizen has a degree of knowledge about the nature, behavior, 
power, and consequences of technology from a broad perspective. Inherently, it involves educational programs 
where learners become engaged in critical thinking as they design and develop products, systems, and environments 
to solve practical problems” (p.250). 

New forms of literacy that have emerged in the new century (such as digital literacy, media literacy, 
information literacy, etc.), and the fact that with technological possibilities learning has become disconnected from 
time and space, make digital transformation inevitable for universities. This transformation may take place in three 
dimensions: administrative, educational and infrastructural. The purpose of the present study is to evaluate the 
program information packages created in one university on the basis of digital standards in the educational 
dimension of undergraduate programs, as per the digital transformation framework of education programs. 

METHOD 

This study used a descriptive case study design to examine higher education curricula. In this study, the 
packages of the undergraduate programs of Yeditepe University, Turkey, were evaluated within the frame of the 
digital university. The case study was chosen in order to “retain the holistic and meaningful characteristics of real- 
life events [including] organizational and managerial processes” (Yin, 1989, cited in Toledo, 2005). This study 
included a program evaluation study and document review of written curricula, in accordance with the “Educational 
Connoisseurship and Criticism Model developed by Eisner, who used the concept of connoisseurship for “promoting 
the ability to see a process or a work from multiple perspectives” (Eisner, 1988, cited in Alkin, 2013). This model 
has been applied in various areas of education, especially real-life curriculum extensions. 

The key to developing knowledge within schooling and other educational settings (such as the family) 
is to create a varied and stimulating environment in which people become ‘immersed’. Educators also 
need to encourage people to try make meaning; to ‘read’ (or conceptualize) the situation. This they do 
by constructing images ‘derived from the material the senses provide’ and refining ‘the senses [as] a 
primary means for expanding ... [one’s own] consciousness’ (Eisner 1994, cited in Smith, 2005). 

While technology has long been a part of educational settings and curriculum, digital technologies and skills have 
only recently started to become an integral part of higher education. Therefore, observing such settings from the 
inside could help us develop more realistic interpretations. According to Eisner (1985), teaching and learning, like 
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art, require qualitative judgements, are often unpredictable, are non-prescriptive and can be performed with ‘such 
skill and grace that for the student as well as for the teacher the experience can justifiably be called aesthetic’ 
(Kelehear, 2010). According to Ozkan and McKenzie (2006) the connoisseurship model has two major aspects in 
evaluating educational technologies: a holistic approach to the analysis and interpretation of data, and multiple 
perspectives towards the evaluative tasks. In this research all faculties and curricula were examined within such a 
holistic approach. 

Research Questions 

Program learning outcomes, course goals, course topics, and teaching and assessment methods for all the 
relevant program packages were taken into consideration as evaluation content. Regarding the evaluation criteria, a 
form was prepared (described in more detail in the section on data collection) according to ISTE standards. 

The study’s research question and sub-questions were defined as follows: ‘What is the level of 
digital/technological competence of the undergraduate programs in Yeditepe University?’ 

1. What is the level of digital/technological competence of the general learning outcomes of the programs? 

2. What is the level of digital/technological competence of the goals, contents, teaching methods and forms of 
assessment of the courses included in the programs? 


Sample 


Yeditepe University, founded in 1996 as a private University, carries out its academic mission through 12 
faculties with 60 programs: Medicine, Dentistry, Pharmacy, Health Science, Law, Engineering and Architecture, 
Education, Arts and Sciences, Economics and Administrative Sciences, Commerce, Communication, and Fine Arts. 
Yeditepe University is the first foundation university in Turkey accredited by the Institutional Accreditation 
Program of the Association of European Universities (EUA) (http://www.yeditepe.edu.tr/about-yeditepe/general- 
information). In the 2013-2014 academic year, there was a total of 2,344 programs. The information packages of all 
of the undergraduate programs of Yeditepe University were included within the scope of the evaluation (N=21) 
although the number of courses provided in every program varied, a total of 565 courses were evaluated. Only field- 
specific courses for each faculty were taken into consideration. 


Table 1. Distribution of faculty departments and number of courses 


Faculty 

Number of departments 

Number of field-specific courses 

Pharmacy 

1 

61 

Education 

3 

59 

Arts and Sciences 

4 

103 

Economics and Administrative Sciences 

4 

91 

Fine Arts 

3 

108 

Engineering and Architecture 

4 

102 

Commerce 

2 

41 

TOTAL 

21 

565 


Data Collection Tool and Evaluation Process 

The data collection tool used to evaluate the program information packages was based on ISTE 
(International Society for Technology in Education) 2007 standards. ISTE is considered the premier source for 
standards, performance indicators and outcomes for the assessment of ICT digital literacy competencies. It has been 
adopted by multiple institutions and K-12 commercial assessment developers (CETF ICT Digital Literacy Initiative, 
Consensus Document, 2008). ISTE asserts that merely using technology is no longer sufficient because today’s 
students need to be capable of utilizing technology for analysing, learning and researching. The skills of the digital 
age are crucial for the students’ preparation for working and general life, and their future contribution to society. 
ISTE lists its standards concerning the skills and knowledge that students need to attain in an increasingly global and 
digital world in seven categories: Creativity and Innovation, Communication and Collaboration, Research and 
Information Fluency, Critical Thinking, Problem Solving and Decision Making, Digital Citizenship, Technology 
Operations and Concepts (ISTE, 2007). 
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The evaluation that was carried out at the level of program outcomes and courses was limited by the 
information provided in the program information packages. The evaluation attempted to identify the standards used 
in the evaluation form in the following categories of the program information packages: 

• Embedding new skills and new literacies into program outcomes 

• Curriculum and general aims of courses 

• Course content 

• Teaching methods 

• Assessment methods 

The above categories include the most basic elements that constitute a curriculum. Among ISTE standards, 
those that are relevant for the evaluation of program information packages were determined in order to prepare a 
“program evaluation rubric” according to ISTE standards. This included the six criteria concerning each of the 
topics of the program learning outcomes, course goals and course content, and one question concerning teaching and 
assessment methods. The rubric also included a total of 20 questions, evaluated at 3 levels (yes, partially, no). The 
evaluation was carried out in three stages. Firstly, a curriculum specialist and an instructional technology specialist 
compared the programs to ISTE standards. Second, the percentage compliance with the checklists for each criteria 
was determined. To ensure the reliability of the rubrics, the inter-rater agreement was checked, using the Weighted 
Kappa coefficient, where “Kappa Agreement < 0 indicates less than chance agreement, 0.01-0.20 indicates slight 
agreement, 0.21- 0.40 indicates fair agreement, 0.41-0.60 indicates moderate agreement, 0.61-0.80 indicates 
substantial agreement, and 0.81-0.99 indicates almost perfect agreement” (Viera, Garrett, 2005 p. 362). 


Table 2. Kappa Agreement coefficients in terms of ISTE standarts 



Program outcomes 

Course goals 

Course content 

Teaching 

Assessment 

Weighted Kappa coefficient 

80.3* 

74.4* 

60.0** 

72.2* 

68.5* 

Course Number 

565 

565 

565 

565 

565 


** = p< .01 


The weighted Kappa coefficients for the inter-rater reliability are presented in Table 2, which shows that 
the level of agreement between the two raters was substantial agreement except for the “course content” dimension. 

RESULTS 

Table 3 presents the results concerning the learning outcomes and teaching and assessment methods 
included in the information packages of 20 programs examined according to ISTE criteria. 

As mentioned before in the data collection section, there were six questions regarding each program’s 
learning outcomes, course goals and course contents. There was one question each for the teaching and assessment 
methods categories. ‘Yes’ answer was coded as ‘2’, ‘partially’ as ‘1’, and ‘No’ as ‘O’. The total score for each 
category was then transformed into a percentile based on the minimum and maximum values, as shown in Table 3. 
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Table 3. Technological competence levels of the categories in Program Information Packages. 


Program 

Integration Level 
of the Outcomes of 

Integration Level 
of the General 

Integration Level 
of the Courses 

Integration Level 
of the Teaching 

Integration Level 
of the Assessment 


the Program 

aims of courses 

Content 

Methods 

Methods 

Faculty of Economics 
and Administrative 
Sciences (N=4) 

36% 

11 % 

11 % 

0% 

0% 

Business Administration 

42 % 

25 % 

25 % 

0% 

0% 

Public Administration 

0% 

0% 

0% 

0% 

0% 

Political Science and 
International Relations 

42 % 

0% 

0% 

0% 

0% 

Business Informatics 

58% 

17 % 

17 % 

0% 

0% 

Faculty of Engineering 
and Architecture 
(N=3) 

50% 

38 % 

42 % 

42 % 

42 % 

Computer Engineering 

67% 

42 % 

58 % 

50% 

50% 

Biomedical Engineering 

50% 

25 % 

25 % 

50% 

50% 

Industrial and Systems 
Engineering 

50% 

50% 

50% 

50% 

50% 

Food Engineering 

50% 

33 % 

33 % 

0% 

0% 

Faculty of Pharmacy 
(N=l) 

50% 

42 % 

42 % 

50% 

50% 

Faculty of Arts and 
Sciences (N=4) 

28% 

26% 

27 % 

25 % 

0% 

Translation and 
Interpreting Studies 

25 % 

33 % 

33 % 

50% 

0% 

Mathematics 

42 % 

42 % 

33 % 

50% 

0% 

Sociology 

10% 

17 % 

17 % 

0% 

0% 

History 

33 % 

10% 

25 % 

0% 

0% 

Faculty of Fine Arts 
(N=3) 

22% 

17 % 

28 % 

33 % 

17 % 

Plastic Arts 

0% 

17 % 

17 % 

0% 

0% 

Theatre 

33 % 

17 % 

33 % 

50% 

50% 

Art Management 

33 % 

17 % 

33 % 

50% 

0% 

Faculty of Commerce 
(N=2)' 

54% 

26% 

26% 

0% 

0% 

Management 

Information Systems 

58% 

42 % 

42 % 

0% 

0% 

International Finance 

50% 

10% 

10% 

0% 

0% 

Faculty of Education 
(N=3)' 

44% 

44 % 

47 % 

50% 

33 % 

Guidance and 
Psychological 
Counselling 

50% 

50% 

42 % 

50% 

0% 

Teaching Mathematics 

58% 

50% 

50% 

50% 

50% 

English Language 
Teaching 

25 % 

33 % 

48 % 

50% 

50% 


Results Concerning the First Sub-Question 

The first sub-question of the study was ‘What is the level of digital/technological competence of the 
general learning outcomes of the programs?’ Table 2 shows that the percentage departmental technological 
competence scores for the learning outcomes ranged between 58 % and 25 %, except for Public Administration, 
which, at 0 %, can be assumed to be an outlier. This indicates that the programs are at least considering the objective 
of educating technologically literate individuals, capable of utilizing technology, both personally and professionally. 
For instance, the learning outcomes of the pharmacology program included the learning outcome of ‘Keeping 
abreast of and being equipped to apply the fundamental pharmaceutical and health science principles which guide 
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future developments in modem technology, and being able to share these experiences and skills with colleagues 
through participation in professional development education seminars’. This is clearly aimed at developing 
individuals that can effectively use modern technology in their professional development. Similarly, the following 
learning outcome of the Psychological Counselling and Guidance Program, ‘Learning how to use internet and 
technology for accessing information, sharing information, professional development and data analysis’, emphasized 
that effective use of technology and internet is an important part of professional development. In the mathematics 
program, on the other hand, the learning outcome concerning technology, ‘Ability to learn, choose and use 
necessary information technologies’, was limited to information technologies. Overall, the results indicate that 
technology in the university under study is considered as a forward-looking tool for ensuring life-long learning and 
professional development for those that graduate from these programs. 

In some programs, technology is defined as one of the key professional competencies of the program. For 
instance, one of the learning outcomes of the Translation and Interpreting Studies program, ‘Being able to use 
current translation technologies to do research and to reach resources’, concerned a technology (translation 
technology) that is directly related with the students’ future occupation. Learning outcomes configured in this way 
should have corresponding elements within the program. 

Results Concerning the Second Sub-Question 

The second sub-question of the study asked ‘What is the level of digital/technological competence of the 
goals, contents, teaching methods and assessment ways of the courses included in the programs?’ The way that these 
competences were included in the program goals can be clarified by examining three dimensions: course goals, 
contents, educational statuses and assessment. Analysis of the courses, as one of the most important tools for 
developing the qualities generally aimed for by the programs, in terms of the ISTE standards showed that they were 
not adequate (ranging from 50 % down to 10 %) to develop individuals that can effectively utilize technology 
personally and professionally, as expressed in program learning outcomes. For example, although objectives such as 
‘Students should be able to effectively utilize computer and information technologies commonly-used in the social 
sciences’ and ‘Students will prove their understanding of the rapidly-evolving dynamics of national and global 
environments requires constant self-assessment, life-long learning, and the ability to formulate innovative solutions 
to maintain their personal and professional development’ were included in the program outcomes of the department 
of political sciences, none of the courses in the program included any course goal or content to enable students to 
achieve these outcomes. 

Similarly, the program outcomes of the department of Food Engineering included two goals related to 
digital competency: ‘Usage of information technologies to solve food engineering problems and to design processes’ 
and ‘Appreciation of the need for lifelong learning, and for follow up on and application of developments in science 
and technology’. However, there was only one course related to the first of these digital competency goals, while 
there was no course or course content related to the second goal. Similar examples could be given for other 
departments. 


CONCLUSION 

In conclusion, the information packages of the undergraduate programs of Yeditepe University, examined 
here as a case, generally reflect intentions for integrating technology into educational programs. However, these 
intentions are not yet very manifested in the programs’ curricula, particularly in the learning outcomes. Examination 
the categories of course goals, contents, teaching methods and assessment processes shows that they do not 
adequately support the goal of developing individuals that can effectively use technology as stated in program 
learning outcomes. Technology therefore needs to be utilized as a more effective educational tool, particularly in the 
creation of course contents and configuration of the learning process. Although it would be unrealistic to expect the 
immediate and direct inclusion of the current technologies specific to every field and occupation in the related 
education programs (other than in a few exceptional cases), at least the educational processes that aim to create 
awareness of new technologies should be configured according to clear standards, while innovative learning 
resources should also be developed. 
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In order for a university curriculum to have a continuously developing structure capable of fulfilling 
today’s changing needs, it is necessary to ensure that it is technologically friendly and interdisciplinary. At this 
point, big data and new data analysis techniques can help us in curriculum mapping. ‘Big data denotes datasets that 
are large, complex, and difficult to store, search, share, analyse, and visualize using commonly available analytical 
tools and methods’ (Expanding Evidence Approaches for Learning in a Digital World U.S. Department of Education 
Office of Educational Technology Report, 2013, p.7). Utilization of digital analysis and planning techniques by 
universities as tools for regulating education would be useful. It should also be kept in mind that ‘curriculum design 
is not carried out in isolation but forms part of an iterative cycle of market research, planning, development, 
implementation and review. However, within some institutions, there may be barriers to designing curricula that 
meet the needs of lifelong learners’ (Drysdale, 2012, p. 20). 

Finally, as indicated by the results of this study, although technology has entered into all fields and 
programs, at least regarding learning outcomes, it is not yet adequately represented in the process of learning 
Therefore, it needs to be ensured that curricula based on digital literacy competencies prepared according to clear 
standards are established and evaluated. If such curricula that are founded on digital literacy are established then 
academic competencies and the quality of learning processes will improve. This would allow digital curricula to be 
revised repeatedly, quickly and economically. It would be valuable for a future study of this issue to make an in- 
depth evaluation by interviewing faculty members and students regarding the way their actual courses progress. 
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